Screenshot file patterns (closes #1602, #1651, #1974, #1975) #2086

thireven · 2018-01-30T22:13:38Z

Starting this PR to begin discussions about this as well as address an issue I have with the current three test suites: I'm not able to pass all the tests from a fresh clone of testcafe. I'm on OSX 13.3.3 running Node v9.2.0. All three test suites fail to complete. I can create a separate bug issue for this if necessary.

I still need to write test cases for each of these patterns which I hope to start on after resolving my issue above.

Here is the documentation I've written so far for this new feature:

Patterns

Pattern	Description
`${FIXTURE}`	Fixture name
`${TEST}`	Test name
`${TEST_INDEX}`	Test index
`${FILE_INDEX}`	Numbers a file in NNN format for files of the same name in the same folder
`${DATE}`	Date the test was run (YYYY-MM-DD)
`${TIME}`	Time the test was run (HH-mm-ss)
`${DTF_<FORMAT>}`	Custom DateTime Format. Any supported MomentJS display format can be used here
`${USERAGENT}`	Concatenation of `${BROWSER}`, `${BROWSER_VERSION}`, `${OS}`, and `${OS_VERSION}`
`${BROWSER}`	Name of browser of current test
`${BROWSER_VERSION}`	Browser version of current test
`${OS}`	Name of operating system current test is currently run on
`${OS_VERSION}`	Version of operating system current test is currently run on
`${QUARANTINE_ATTEMPT}`	By default "1" (if you add it to the template but run tests without the `-q` option)

Usage & Examples

Set the pattern to be used with the -p or --screenshot-path-pattern flag.

testcafe chrome test1.js -s screenshots -p ${DATE}_${TIME}/test-${TEST_INDEX}/${USERAGENT}/${TEST_INDEX} (this pattern is the default one if no pattern is specified for backwards compatibility)

The pattern above reproduces the default file structure for the screenshot files:

└── 2018-01-30_13-32-47
    ├── test-1
    │   └── Chrome_63.0.3239_Mac_OS_X_10.13.3
    │       ├── 1.png
    │       └── thumbnails
    │           └── 1.png
    └── test-2
        └── Chrome_63.0.3239_Mac_OS_X_10.13.3
            ├── 2.png
            └── thumbnails
                └── 2.png

Patterns can also be used within the takeScreenshot() method like so: takeScreenshot('${TEST}/${OS}/${OS_VERSION}/${FILE_INDEX}'). Note that specifying a pattern in takeScreenshot() will override the pattern passed in the CLI for that specific screenshot.

Examples of patterns and their output:

${DATE}_${TIME}/${BROWSER}_${FILE_INDEX}_${FIXTURE}_${TEST}_${TEST_INDEX}

├── 2018-01-30_13-39-15
│   ├── Chrome_001_Screenshots-Pattern_Huehue_2.png
│   ├── Chrome_001_Screenshots-Pattern_Test_1.png
│   ├── Chrome_002_Screenshots-Pattern_Huehue_2.png
│   ├── Chrome_002_Screenshots-Pattern_Test_1.png
│   ├── Chrome_003_Screenshots-Pattern_Test_1.png
│   └── thumbnails
│       ├── Chrome_001_Screenshots-Pattern_Huehue_2.png
│       ├── Chrome_001_Screenshots-Pattern_Test_1.png
│       ├── Chrome_002_Screenshots-Pattern_Huehue_2.png
│       ├── Chrome_002_Screenshots-Pattern_Test_1.png
│       └── Chrome_003_Screenshots-Pattern_Test_1.png

${FIXTURE}/${TEST}_${FILE_INDEX}

└── Screenshots-Pattern
    ├── Huehue_001.png
    ├── Huehue_002.png
    ├── Test_001.png
    ├── Test_002.png
    ├── Test_003.png
    └── thumbnails
        ├── Huehue_001.png
        ├── Huehue_002.png
        ├── Test_001.png
        ├── Test_002.png
        └── Test_003.png

${FIXTURE}/${OS}_${BROWSER}/${TEST}/${FILE_INDEX}

── Screenshots-Pattern
    └── Mac-OS-X_Chrome
        ├── Huehue
        │   ├── 001.png
        │   ├── 002.png
        │   └── thumbnails
        │       ├── 001.png
        │       └── 002.png
        └── Test
            ├── 001.png
            ├── 002.png
            ├── 003.png
            └── thumbnails
                ├── 001.png
                ├── 002.png
                └── 003.png

${DTF_YYYY}/${DTF_MMMM+w}/${DTF_Do of MMM}/${TEST}

── 2018
    └── March+9
        └── 1st-of-Mar
            ├── Huehue.png
            ├── Test-2.png
            ├── Test.png
            └── thumbnails
                ├── Huehue.png
                ├── Test-2.png
                └── Test.png

testcafe-build-bot · 2018-01-30T22:36:55Z

❌ Tests for the commit 82c465f have failed. See details:

AlexanderMoskovkin · 2018-02-02T12:35:06Z

Hi @thireven,

Thank you for your contribution, we appreciate it a lot. Your proposal and current achievements look impressing!

IMPORTANT: The usage of %FILENUMBER% is important since screenshots will overwrite each other if they have the same name structure. %FILENUMBER% would automatically increment the name of the file to prevent such scenarios.

Maybe we shouldn't overwrite screenshots in this case because it can be unexpected for a person. I guess we can just add a numeral postfix for a file name in this case.

/cc @VasilyStrelyaev @DevExpress/testcafe Please take a look at this proposal, do you have some comments?

I have with the current three test suites: I'm not able to pass all the tests from a fresh clone of testcafe. I'm on OSX 13.3.3 running Node v9.2.0. All three test suites fail to complete.

Could you please clarify what errors do you have. Meanwhile we'll try to run in on our side with the same environment.

Also we have a "useragent" dependency in the project. You can use it to parse user agent.

VasilyStrelyaev · 2018-02-02T14:49:13Z

src/cli/argument-parser.js

@@ -89,6 +89,7 @@ export default class CLIArgumentParser {
            .option('-b, --list-browsers [provider]', 'output the aliases for local browsers or browsers available through the specified browser provider')
            .option('-r, --reporter <name[:outputFile][,...]>', 'specify the reporters and optionally files where reports are saved')
            .option('-s, --screenshots <path>', 'enable screenshot capturing and specify the path to save the screenshots to')
+            .option('-p, --pattern <pattern>', 'screenshot files are named using specific patterns: %BROWSER%, %BROWSERVERSION%, %OS%, %OSVERSION%, %USERAGENT%, %DATE%, %TIME%, %FIXTURE%, %TEST%, %TESTNUMBER%, %FILENUMBER%')


Maybe

use patterns to compose screenshot file names and paths: %BROWSER%, %BROWSERVERSION%, %OS%, %OSVERSION%, %USERAGENT%, %DATE%, %TIME%, %FIXTURE%, %TEST%, %TESTNUMBER%, %FILENUMBER%'

AndreyBelym · 2018-02-02T15:52:04Z

Unfortunately, we can't use %VARIABLE% syntax for placeholders, because Windows cmd.exe uses it for interpolating environment variables. Moreover, %DATE% and %TIME% are predefined variables. Also, rules of escaping aren't obvious (you need to put escaping character - ^ - not only before % signs, but also before quotes):

D:\testcafe-phoenix>node -e console.log(process.argv[1]) %DATE%      
Fri                                                                  
                                                                     
D:\testcafe-phoenix>node -e console.log(process.argv[1]) "%DATE%"    
Fri 02/02/2018                                                       
                                                                     
D:\testcafe-phoenix>node -e console.log(process.argv[1]) "^%DATE^%"  
^%DATE^%                                                             
                                                                     
D:\testcafe-phoenix>node -e console.log(process.argv[1]) ^"^%DATE^%^"
%DATE%

I think, we have to use shell-neutral template syntax, like Mustache. I would like to have standard JS template strings syntax: ${VAR}, but it will interfere with variable substitution on Unix shells (but for them you can just put single quotes around the pattern to disable substitution).

thireven · 2018-02-02T18:45:10Z

Could you please clarify what errors do you have. Meanwhile we'll try to run in on our side with the same environment.

From gulp test-server:

  1) CLI argument parser Should use "test" and "tests" dirs if source files are not specified:

      AssertionError: expected [ Array(4) ] to deeply equal [ Array(2) ]
      + expected - actual

       [
         "/Users/tnguyen/Projects/tc_test/test/server/data/file-list/test/test-dir-file.js"
         "/Users/tnguyen/Projects/tc_test/test/server/data/file-list/tests/tests-dir-file.js"
      -  "/Users/tnguyen/Projects/tc_test/test/server/data/file-list/Test/test-dir-file.js"
      -  "/Users/tnguyen/Projects/tc_test/test/server/data/file-list/Tests/tests-dir-file.js"
       ]

      at test/server/cli-argument-parser-test.js:308:36
      at <anonymous>

From gulp test-functional-local:

Running tests in browsers: chrome, ie, firefox
  1) "before all" hook

  0 passing (7s)
  1 failing

  1)  "before all" hook:
     TypeError: Cannot read property 'path' of null
      at checkBrowserPath$ (node_modules/testcafe-browser-tools/lib/api/open.js:34:33)
      at tryCatch (node_modules/testcafe-browser-tools/node_modules/babel-runtime/regenerator/runtime.js:72:40)
      at Generator.invoke [as _invoke] (node_modules/testcafe-browser-tools/node_modules/babel-runtime/regenerator/runtime.js:334:22)
      at Generator.prototype.(anonymous function) [as next] (node_modules/testcafe-browser-tools/node_modules/babel-runtime/regenerator/runtime.js:105:21)
      at tryCatch (node_modules/testcafe-browser-tools/node_modules/babel-runtime/regenerator/runtime.js:72:40)
      at invoke (node_modules/testcafe-browser-tools/node_modules/babel-runtime/regenerator/runtime.js:146:20)
      at bound (domain.js:280:14)
      at runBound (domain.js:293:12)
      at node_modules/testcafe-browser-tools/node_modules/babel-runtime/regenerator/runtime.js:191:11
      at new Promise (<anonymous>)

From gulp test-client:

[10:41:00] Starting 'fast-build'...
[10:41:00] Finished 'fast-build' after 31 μs
[10:41:07] Finished 'worker:lint' after 14 s
[10:41:07] Finished 'lint' after 15 s
[10:41:07] Starting 'build'...
[10:41:07] Finished 'build' after 10 μs
[10:41:07] Starting 'test-client'...
QUnit server listens on http://localhost:2000

It just hangs after the QUnit line and doesn't do anything else.

AlexanderMoskovkin · 2018-02-05T15:58:32Z

So according to @AndreyBelym's proposal the template will look like

testcafe chrome test1.js -s screenshots -p "${DATE}_${TIME}/test-${TESTNUMBER}/${USERAGENT}/${TESTNUMBER}"

How do you like this?
/cc @thireven @VasilyStrelyaev @DevExpress/testcafe

thireven · 2018-02-05T17:29:12Z

test1.js -s screenshots -p >"${DATE}_${TIME}/test-${TESTNUMBER}/${USERAGENT}/${TESTNUMBER}"

I'm fine with that. Was also trying out using # as the delimiter over the weekend as well and it seems to work for OSX and Windows without any issues.

testcafe-build-bot · 2018-02-05T20:33:43Z

❌ Tests for the commit a31dc1b have failed. See details:

miherlosev · 2018-02-06T08:12:36Z

I don't like this way.
Template string with separators has the following cons:

difficult to use conditional logic,
difficult to format datetime and useragent
potential problems with path separator char (https://nodejs.org/api/path.html#path_path_sep).

I propose to use a single function (packed as module) to build a custom screenshot path.
The function was named as CustomScreenshotPathProvider.

Example:

testcafe chrome -s screenshots -p OrganizationShare.js
testcafe ie -s screenshots -p DevFTP.js

Where OrganizationShare.js

/*Import necessary modules*/

export default function (runnerInfo, userAgent) {
    /*Custom logic*/
    return path.combine(runnerInfo.fixtureName, runnerInfo.testName, userAgent.browser...)
}

Pros of this way:

complex pattern is incapsulated to file with human-readable alias.
flexible and power (allows to use all nodejs modules)

What do you think about it?

AlexanderMoskovkin · 2018-02-06T13:36:08Z

difficult to use conditional logic,

What conditional logic would you like in screenshot path template. I can't imagine a real-life case for that

potential problems with path separator char

It's not a problem. We normalize it by using path. We use the same way for test file path, screenshot dir path etc..

difficult to format datetime and useragent

There is not problems with useragent since we have ${BROWSER}, ${BROWSERVERSION}, ${OS}, ${OSVERSION}.

I see the only restriction here is it's impossible to format date. I'm not sure it's too important here. (if necessary we can add something like ${mm}, ${yyyy} later).

I feel the way with a separated js file is to complex and less native. I think it's not too nice to add a separated settings file to such specific feature. It possible we'll add a general settings file in future.

But I guess we should add such function to the programming api (is the same way as we did for runner.src). With this you can realize any logic there.

AlexanderMoskovkin · 2018-02-06T13:45:23Z

(UPDATED):

Additionally we already have some kind of templating for takeScreenshot action. See the Take Screenshot topic (UPD: I missed, actually we don't have. It just an example).

I guess we can just extend it for a whole test run (and avoid a breaking change there). So the API can see in the following way:

// CLI
testcafe chrome test.js -s <path> --screenshot-template "{currentDate}/test-{testIndex}/{userAgent}/{screenshotIndex}.png"

Available options:
- { date }
- { time }
- { testIndex }
- { screenshotIndex }
- { quarantineAttempt }

- { fixture }
- { test }
- { userAgent }
- { browser }
- { browserVersion }
- { os }
- { osVersion }

// Programming
runner.screenshot(path [, takeOnFails][, filePath])
// where filePath is a template string or a function that take the info and returns a value

How do you like this? /cc @VasilyStrelyaev @thireven @DevExpress/testcafe

miherlosev · 2018-02-07T07:58:50Z

What conditional logic would you like in screenshot path template. I can't imagine a real-life case for that

I don't want to invent user cases.
You option from previous comment covers known cases.
So, let it be.

AlexanderMoskovkin · 2018-02-07T09:19:11Z

Additionally we already have some kind of templating for takeScreenshot action. See the Take Screenshot topic.

My bad. We don't have templating there yet. It's just a default value example in the Take Screenshot Topic. I've updated my comment a little bit. But the proposed API is still actual

thireven · 2018-02-09T19:34:30Z

Additionally we already have some kind of templating for takeScreenshot action. See the Take Screenshot topic.

Going off of what's there, would that mean that the pattern would change to {...} instead of ${...}? Would that run into any issues on various platforms?

AlexanderMoskovkin · 2018-02-14T13:20:32Z

Hi @thireven,

After some considetaion let's stay with the following API:

-p or --screenshot-path-pattern

testcafe chrome test.js -s <path> -p "${DATE}_${TIME}/test-${TEST_INDEX}/${USERAGENT}/${FILE_INDEX}.png"

Available options:
- ${DATE}
- ${TIME}
- ${TEST_INDEX}
- ${FILE_INDEX}
- ${QUARANTINE_ATTEMPT} --by default "1" (if you add it to the template but run tests without the `-q` option).

- ${FIXTURE}
- ${TEST}
- ${USERAGENT}
- ${BROWSER}
- ${BROWSER_VERSION}
- ${OS}
- ${OS_VERSION}

If TestCafe creates a screenshot when a test fails (--screenshots-on-fails option enabled) we just add the -error postfix to the filename.

testcafe-build-bot · 2018-03-01T20:42:17Z

❌ Tests for the commit 715b42f have failed. See details:

thireven · 2018-03-02T01:23:24Z

Made some updates according to what @AlexanderMoskovkin posted and added support for custom datetime formatting (DTF). The original PR info has been updated with information on DTF and an example pattern and output.

On the testing side, I'm still struggling to get two of them to succeed. gulp test-server now works and passes, but I still have the same issues with the other two.

testcafe-build-bot · 2018-03-02T01:47:03Z

❌ Tests for the commit 48ee194 have failed. See details.

testcafe-build-bot · 2018-03-07T12:38:34Z

✅ Tests for the commit 48ee194 have passed. See details.

testcafe-build-bot · 2018-03-07T14:51:20Z

❌ Tests for the commit 48ee194 have failed. See details:

AlexKamaev · 2018-03-07T15:27:55Z

src/screenshots/capturer.js

        this.quarantineAttemptNum = namingOptions.quarantineAttemptNum;
        this.testIndex            = namingOptions.testIndex;
        this.screenshotIndex      = 1;
        this.errorScreenshotIndex = 1;

-        var testDirName     = `test-${this.testIndex}`;
-        var screenshotsPath = this.enabled ? joinPath(this.baseScreenshotsPath, this.baseDirName, testDirName) : '';
+        var screenshotsPath = this.enabled ? this.baseScreenshotsPath : '';


this change leads to failures in test, see comment

AlexKamaev · 2018-03-07T15:55:21Z

Hi

As you are using OSX, you do not have IE. But in the log I see the following:
Running tests in browsers: chrome, ie, firefox
and
at checkBrowserPath$ (node_modules/testcafe-browser-tools/lib/api/open.js:34:33)

So, I think it should work if you remove IE (and other missing browsers) in
..\test\functional\config.js

Simply modify the testingEnvironments[testingEnvironmentNames.localBrowsers] variable value

As for the gulp test-client, these tests work in a browser and it is required to open http://localhost:2000 in your browser.

I have also noticed that some AppVeyour tests failed.

145  1) [API] t.takeScreenshot() Should take a screenshot:
146
147      AssertionError: expected false to deeply equal true
148      + expected - actual
149
150      -false
151      +true
152      
153      at test\functional\fixtures\api\es-next\take-screenshot\test.js:18:88
154      at <anonymous>

Please take a look at this test.

it('Should take a screenshot', function () {
	return runTests('./testcafe-fixtures/take-screenshot.js', 'Take a screenshot', { setScreenshotPath: true })
		.then(function () {
			expect(SCREENSHOT_PATH_MESSAGE_RE.test(testReport.screenshotPath)).eql(true);  <---- failed here because of changes in comment
			expect(assertionHelper.checkScreenshotsCreated(false, 4)).eql(true);
		});
});

I have added a comment to the problematic code in the 'Files changed' tab

testcafe-build-bot · 2018-03-26T20:24:05Z

❌ Tests for the commit c5eeefe have failed. See details:

…vExpress#1974, DevExpress#1975)

Patterns that generate the same filename will now be auto-incremented instead of being overwritten.

…arsing logic to its own file

…assuming the old format

testcafe-build-bot · 2018-06-09T14:26:12Z

❌ Tests for the commit 97d9034 have failed. See details:

testcafe-build-bot · 2018-06-18T11:23:40Z

❌ Tests for the commit 77f4ddf have failed. See details:

testcafe-build-bot · 2018-06-19T11:06:49Z

❌ Tests for the commit 6999614 have failed. See details:

miherlosev · 2018-06-19T12:37:16Z

This PR has intersections of the files changed with #2476.
I will continue to work after merge #2476.

miherlosev · 2018-07-04T11:12:43Z

It will take time to resolve all the conflicts. It'll be more convenient to create a separate PR.
So, I will close this PR.

MargaritaLoseva · 2018-07-12T13:37:54Z

I'm working on this

VasilyStrelyaev reviewed Feb 2, 2018

View reviewed changes

AlexanderMoskovkin mentioned this pull request Feb 8, 2018

Adding screenshot and other info to test case name means it complicates statistics DevExpress/testcafe-reporter-xunit#2

Closed

AlexanderMoskovkin mentioned this pull request Feb 20, 2018

Support screenshot path pattern #2152

Closed

AlexKamaev closed this Mar 7, 2018

AlexKamaev reopened this Mar 7, 2018

AlexKamaev reviewed Mar 7, 2018

View reviewed changes

thireven added 3 commits June 9, 2018 16:30

Screenshot file patterns (closes DevExpress#1602, DevExpress#1651, De…

c487d86

…vExpress#1974, DevExpress#1975)

Updated pattern to use ${...} format.

470bb5e

Patterns that generate the same filename will now be auto-incremented instead of being overwritten.

Merge fixes. Renamed a few patterns. Added QUARANTINE_ATTEMPT pattern

a738576

thireven added 2 commits June 9, 2018 16:37

Added support for custom datetime format pattern. Moved all pattern p…

8d6e792

…arsing logic to its own file

Updated screenshot test to use passed in screenshot paths instead of …

97d9034

…assuming the old format

miherlosev force-pushed the screenshot-patterns branch from c5eeefe to 97d9034 Compare June 9, 2018 13:45

rewrite

77f4ddf

fix tests

6999614

AndreyBelym added this to the Sprint #13 milestone Jun 21, 2018

AndreyBelym assigned miherlosev Jun 21, 2018

miherlosev closed this Jul 4, 2018

miherlosev mentioned this pull request Jul 12, 2018

Describe new features #2615

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Screenshot file patterns (closes #1602, #1651, #1974, #1975) #2086

Screenshot file patterns (closes #1602, #1651, #1974, #1975) #2086

thireven commented Jan 30, 2018 •

edited

Loading

testcafe-build-bot commented Jan 30, 2018

AlexanderMoskovkin commented Feb 2, 2018

VasilyStrelyaev Feb 2, 2018

AndreyBelym commented Feb 2, 2018 •

edited

Loading

thireven commented Feb 2, 2018

AlexanderMoskovkin commented Feb 5, 2018

thireven commented Feb 5, 2018

testcafe-build-bot commented Feb 5, 2018

miherlosev commented Feb 6, 2018 •

edited

Loading

AlexanderMoskovkin commented Feb 6, 2018

AlexanderMoskovkin commented Feb 6, 2018 •

edited

Loading

miherlosev commented Feb 7, 2018

AlexanderMoskovkin commented Feb 7, 2018

thireven commented Feb 9, 2018

AlexanderMoskovkin commented Feb 14, 2018

testcafe-build-bot commented Mar 1, 2018

thireven commented Mar 2, 2018

testcafe-build-bot commented Mar 2, 2018

testcafe-build-bot commented Mar 7, 2018

testcafe-build-bot commented Mar 7, 2018

AlexKamaev Mar 7, 2018

AlexKamaev commented Mar 7, 2018

testcafe-build-bot commented Mar 26, 2018

testcafe-build-bot commented Jun 9, 2018

testcafe-build-bot commented Jun 18, 2018

testcafe-build-bot commented Jun 19, 2018

miherlosev commented Jun 19, 2018

miherlosev commented Jul 4, 2018 •

edited

Loading

MargaritaLoseva commented Jul 12, 2018

Screenshot file patterns (closes #1602, #1651, #1974, #1975) #2086

Screenshot file patterns (closes #1602, #1651, #1974, #1975) #2086

Conversation

thireven commented Jan 30, 2018 • edited Loading

Patterns

Usage & Examples

Examples of patterns and their output:

testcafe-build-bot commented Jan 30, 2018

AlexanderMoskovkin commented Feb 2, 2018

VasilyStrelyaev Feb 2, 2018

Choose a reason for hiding this comment

AndreyBelym commented Feb 2, 2018 • edited Loading

thireven commented Feb 2, 2018

AlexanderMoskovkin commented Feb 5, 2018

thireven commented Feb 5, 2018

testcafe-build-bot commented Feb 5, 2018

miherlosev commented Feb 6, 2018 • edited Loading

AlexanderMoskovkin commented Feb 6, 2018

AlexanderMoskovkin commented Feb 6, 2018 • edited Loading

miherlosev commented Feb 7, 2018

AlexanderMoskovkin commented Feb 7, 2018

thireven commented Feb 9, 2018

AlexanderMoskovkin commented Feb 14, 2018

testcafe-build-bot commented Mar 1, 2018

thireven commented Mar 2, 2018

testcafe-build-bot commented Mar 2, 2018

testcafe-build-bot commented Mar 7, 2018

testcafe-build-bot commented Mar 7, 2018

AlexKamaev Mar 7, 2018

Choose a reason for hiding this comment

AlexKamaev commented Mar 7, 2018

testcafe-build-bot commented Mar 26, 2018

testcafe-build-bot commented Jun 9, 2018

testcafe-build-bot commented Jun 18, 2018

testcafe-build-bot commented Jun 19, 2018

miherlosev commented Jun 19, 2018

miherlosev commented Jul 4, 2018 • edited Loading

MargaritaLoseva commented Jul 12, 2018

thireven commented Jan 30, 2018 •

edited

Loading

AndreyBelym commented Feb 2, 2018 •

edited

Loading

miherlosev commented Feb 6, 2018 •

edited

Loading

AlexanderMoskovkin commented Feb 6, 2018 •

edited

Loading

miherlosev commented Jul 4, 2018 •

edited

Loading